Organization Monkey

This post is part of a series, describing the assessments used to develop the Institute for Research Design in Librarianship (IRDL).

Each cohort of IRDL had their own communication characteristics, from one in-person cohort meeting at night to review the day’s notes, using a whiteboard wheeled into a group study area, to another cohort chatting virtually, using a Slack channel. These informal collaborations reinforced learning and facilitated the development of a community of practice. We studied the size and strength of the research networks of each Scholar, using social network analysis, to determine if participating in IRDL influenced the evolution of their personal networks during the IRDL year.

Since we were interested in observing any changes over time, we administered an ego-centric network questionnaire four times during the year-long program. We used the free and open-source survey software customized for social network data collection, EgoWeb 2.0, for this assessment.

We installed the EgoWeb 2.0 software on a server (I used Linode) so that the questionnaire could be administered via the internet. LMU’s IRB reviewed the protocol ahead of Time 1. A week before the workshop began, I sent a personalized email with their individual link to the first questionnaire to each participant, to be completed before the first day of the workshop (Time 1); this questionnaire took about 10 minutes to complete. I released the second questionnaire (Time 2) via email on the last morning of the summer research workshop and gave the participants time to complete it that day. Six months after the workshop ended, I sent the third questionnaire (Time 3) and right after the program completed, at the one-year mark, I administered the last questionnaire (Time 4).

In our results (Kennedy, Kennedy, & Brancolini, 2017), we observed evidence that a small network at the outset expands throughout the year of participating in IRDL. In the figure, a visualization of a participant’s research network during their IRDL year, we see a dense cluster of people at Time 2, with the small triangle icons representing the people the participant met during the workshop and had frequent, reciprocal conversations about their research.

Change in a research network of an IRDL Scholar, over the course of one year

My reflection on the use of this tool for assessing the program
Since this was exploratory research, I wasn’t sure what to expect from the data, or how it would inform the design of the program. It was clear, however, from early on in our analysis that the program design was impacting the networks of the participants by being in a cohort, moving through a process at the same time.

In informal conversations throughout the Summer Research Workshop, some of the participants found like-minded researchers who were using similar data collection methods or working in the same functional area of the library. We encouraged these conversations by frequently offering research-focused ice-breaker questions that one participant could ask another, to help foster communication among the cohort. We also prompted the participants to consider who in the cohort or their workplaces at their home institutions they could talk to about their research. We offered a script/outline that the participants could use to talk to their library directors about their research once the summer workshop was completed.

The participants spontaneously noted throughout the year that being prompted to consider who was in their research networks was impactful for them. I also appreciated that the participants set up their own after-the-workshop communication mechanisms, to stay close to those in their cohort to continue seeking and offering help throughout the year. It has been fun to discover many cross-cohort collaborations, some of which have led to major publications in our field (notably Fullington, West, & Albarillo, 2020). An ideal future research project would be to evaluate the networks of the participants after a set period, after completion of their IRDL year, to note how many relationships persist and find out how they are thinking about those connections.

The cost of this assessment tool
To secure space on a Linode server and hold the small amount of data from the questionnaires cost $5.00 per month. Analysis was completed using R in RStudio Desktop, which are free to use.

Reflections on Practitioner Research: A Practical Guide for Information Professionals. Lee Ann Fullington, Brandon K. West, and Frans Albarillo, eds. Chicago, IL: Association of College and Research Libraries, 2020.

Kennedy, Marie R., David P. Kennedy, and Kristine R. Brancolini. 2017. “The Evolution of the Personal Networks of Novice Librarian Researchers.” portal 17(1): 71-89. https://doi.org/10.1353/pla.2017.0005

Related research, not an assessment of the program, per se. Luo, Lili, Marie R. Kennedy, Kristine R. Brancolini, and Michael Stephens. 2017. “Developing Online Communities for Librarian Researchers: A Case Study.” College & Research Libraries 78(4). https://doi.org/10.5860/crl.78.4.512

Earlier posts in this series:
Introduction post, Confidence scale

This post is part of a series, describing the assessments used to develop the Institute for Research Design in Librarianship (IRDL).

In our 2012 article, we reported on the results of a survey we administered to get a sense of the attitudes, involvement, and perceptions academic librarians had about conducting their own original research. A component of that survey included a multi-part question about confidence in the steps of a research project. Question 10 of the survey looked like this:

Q10. On a scale of 1 to 5, with 1 being “Not at all confident” and 5 being “Very confident,” how would you rate your confidence in performing the following steps in a research project?
[Likert scale 1–5 was presented for each entry: 1 Not at all confident; 2 Slightly confident; 3 Moderately confident; 4 Confident; 5 Very confident]

• • Turning your topic into a question that can be tested
  • Designing a project to test your question
  • Performing a literature review
  • Identifying research partners, if needed
  • Gathering data
  • Analyzing data
  • Reporting results in written format
  • Reporting results verbally
  • Determining appropriate format for disseminating results (poster/presentation/article)
  • Identifying appropriate places to disseminate results (publication/conference)

Most of the respondents rated themselves at points 3 or 4 of this 5-point scale. We understood that this question was a rather blunt measurement and that the steps of a research project were quite more nuanced than as presented in the survey. It did give us, however, enough information about how librarians were assessing their own confidence to know how to move forward in the development of a curriculum for the IRDL program.

We further explored the design of the 10-item scale to measure confidence in the steps of a research project by conducting an exploratory factor analysis. The results of the analysis led us to remove the item, “Identifying research partners, if needed,” and change the wording of the item about performing a literature review. The resulting nine items clustered into three phases: planning, data, and reporting.

Knowing that the nine-item scale lacked the specificity of measurement we desired, we expanded it to 38 items within the three phases. We retained the 5-point Likert scale as measurement. The development of the scale is described in our 2017 article, “The Development and Use of a Research Self-efficacy Scale to Assess the Effectiveness of a Research Training Program for Academic Librarians.” We have used the 38-item scale in subsequent research (Kennedy & Brancolini, 2018) as well as an assessment tool in IRDL. For the purposes of IRDL assessment, in 2022 we condensed the survey to include only 34 items and moved away from the 5-point Likert scale to a 0-100 slider scale, with guiding language of “cannot do at all” near the zero end of the scale, “moderately can do” in the middle, and “highly certain can do” near the 100 end of the scale.

Linked here is the version of the scale in use from 2022-2024.

An example of how the nine-point scale was expanded from the original: The first item on the ten-point scale was “Turning your topic into a question that can be tested.” In the 38-point scale, that item was expanded into three: “turning your topic into a question”; “constructing a question that is reasonable in scope,” and “determining if your research topic makes a contribution to the field.” Expanding just this one item enabled us to measure some practical nuance in the concept of the item.

How the confidence scale was used for IRDL
Research confidence has been shown to be a predictor of research productivity. There is also some evidence that it is a mediating factor between the research training environment and research productivity. To explore the connection between research training and research self-efficacy, we used the scale on the cohorts of IRDL. The scale was delivered to the Scholars as a pre-test (Time 1), in the spring, before their Summer Research Workshop training, and again immediately after completing the training, as a post-test (Time 2).

LMU’s IRB reviewed the protocol ahead of Time 1. At Time 1, I sent an email to all participants of the incoming cohort, to ask them to complete the survey (delivered on the Qualtrics platform), within a two-week time frame. Time 1 was scheduled in the spring, after acceptance into the program but ahead of receiving any program materials (books or access to online course materials). We released the second survey (Time 2) via email on the last morning of the summer research workshop and gave the participants time to complete it that day.

For our purposes, the survey was not anonymized, so that we could compare the differences for each person between Time 1 and Time 2. We calculated average confidence pre-workshop and post-workshop at the group level, change pre/post on each measure, and a change score for each participant. The confidence scores (measured using a t-test) of all participants were statistically higher after the Summer Research Workshop, with a large effect size for each measure. Increasing overall research self-efficacy is intended to increase the probability that the IRDL Scholars will complete their research projects and become accomplished, productive librarian-researchers. Our case study use of the scale confirms that a measurement of research self-efficacy can be a useful tool in assessing the effectiveness of research training for the purpose of improving that training.

My reflection on the use of this tool for assessing the program
This survey took about 5 minutes to complete, so it was convenient for the workshop participants to participate. Capturing their self-assessments before the workshop and immediately after was impactful for us as the developers of this model of continuing education training, to know if the content in the workshop was giving the participants the confidence boost we were hoping to see. It was informative for us to know that the curricular design for the various components of the workshop was providing such a strong increase in confidence for the participants.

The cost of this assessment tool
Because LMU has an institutional subscription to Qualtrics, no grant funds needed to be used for the assessment. No specialized software was needed for the analysis of data.

Kennedy, Marie R. and Kristine R. Brancolini. 2012. “Academic Librarian Research: A Survey of Attitudes, Involvement, and Perceived Capabilities.” College & Research Libraries 73(5): 431-448. https://doi.org/10.5860/crl-276

Brancolini, Kristine R., and Marie R. Kennedy. 2017. “The Development and Use of a Research Self-efficacy Scale to Assess the Effectiveness of a Research Training Program for Academic Librarians.” Library and Information Research 41(124). https://doi.org/10.29173/lirg760

Kennedy, Marie R., and Kristine R. Brancolini. 2018. “Academic Librarian Research: An Update to a Survey of Attitudes, Involvement, and Perceived Capabilities.” College & Research Libraries 79(6). https://doi.org/10.5860/crl.79.6.822

Earlier posts in this series:
Introduction post

Using Albert Bandura’s theory of self-efficacy (1977) as a theoretical base, we designed the Institute for Research Design in Librarianship (IRDL) to address the four main sources of influence on self-efficacy: mastery experiences, social persuasion, modeling, and emotional arousal. We interpreted the component of “mastery experiences” as successfully learning new tasks, providing the participants with the ability to persevere in the face of obstacles. We interpreted “social persuasion” to structured situations in which the participants receive encouragement and experience success in working through challenges. We viewed “emotional arousal” as how people were feeling when they were in the research process. We interpreted “modeling” to allow for viewing the research process of the other Scholars in their cohort, as well as having access to a formal mentor to provide accurate guidance on the research process. Over the course of IRDL we designed multiple points of measurement to assess the efficacy of those components.

To assist us in developing sound assessment mechanisms, in the first year of IRDL (2014) the two program co-investigators (me and Kristine Brancolini) collaborated with two staff members on campus who focused on assessment metrics as part of their regular work. Leading the IRDL assessment effort was the LMU Director of Assessment, Laura Massa, and the LMU Manager of Surveys and Evaluation, Christine Chavez. The four of us mapped the desired outcomes of the program and designed appropriate assessment tools to measure how successful the program model was in meeting the outcomes.

Since we based the design on the components of the Bandura theory, our initial focus of assessment was on them. To determine the possible gains program participants had related to mastery experiences, we used a confidence scale we designed around the components of the research process. To measure possible changes related to social persuasion, we used a personal network series of surveys over the course of the year, to observe the nature of the research-related relationships the participants developed. To determine how the Scholars were self-regulating, we checked in with them via a survey, at the mid-point in the year-long experience. To determine how a mentor-mentee relationship may have affected the participants sense of confidence, we created a survey for the mentors and a separate survey for the participants (the mentees), that was delivered at the end of the program.

With each year of the IRDL program, we had nine separate formalized assessments in place, to help us evaluate if we were meeting our programmatic goals: 1) the confidence scale noted above; 2) a series of research network surveys, noted above; 3) a post-workshop survey; 4) an external review; 5) a pre-/post-workshop research proposal evaluation; 6) a mid-point check-in survey; 7) a summative survey; 8) a mentor feedback survey, noted above; and 9) a scholar feedback survey on the mentor program, noted above.

In a series of blog posts I will focus on each assessment, when it was delivered, what mechanism of data collection was used and its analysis, how much the assessment cost, and a reflection on its effectiveness in measuring what we intended.

I include here the annual schedule, showing the year according to the grant terms, that I used to coordinate the assessments.

I want to put “ask me about my Gantt chart” on a t-shirt someday.

[Edited 8/15/2025 to include the costs of the snacks]

Part of a series of reflections on some of the affective components of the Institute for Research Design in Librarianship (IRDL) program, things I put in place to make the Scholars feel welcomed and delighted in the learning environment.

Candy and snacks of the in-person workshop

As we were planning the first in-person Summer Research Workshop of IRDL we talked about how we would set up the environment for comfort and maximize easy physical interactions for group work and networking. We decided on round tables in the center of the room and two small tables in the back for me and Kris to observe. We decided not to have a lectern or formal panel table at the front of the room; the instructors would have wireless microphones (we had both lapel and hand-held) and could teach throughout the room.

In the back of the room were two couches and a big table between them. I asked the library’s Events Manager, who was partnered with us to make the event as smooth as possible, to load that table up with snacks. There were already going to be planned meals and snacks throughout the days, but in those between times, when someone was maybe feeling like they needed a momentary distraction, I wanted to have snacks available.

The Events Manager came through for me in a big way. She put so many snacks on the table that it was like a party. She bought clear glass candy jars with silver metal lids, candy scoops, and filled them with individually wrapped snacks (Nerds, gummy bears, trail mix) and bulk snacks (peanut M&Ms, etc.). She even brought fresh whole pieces of fruit to mix in.

Throughout the days of the workshop we’d say, “Meet me at the snacks,” if we wanted to chat about something during a break. During the tougher cognitive parts of the workshop, we’d notice more frequent trips of the Scholars to the back of the room to fuel themselves.

Clear glass candy jar with a metal lid, filled with peanut M&Ms and a candy scoop

Starburst, fruit gels, Haribo

I estimate that snacks for the in-person workshop cost around $750 per year. The first year would have cost more, to include the purchase of the candy jars and scoopers. Grant funds were used until a change in allowable costs for this particular grant, restricting being able to pay for food. At that point the library absorbed the cost.

Candy and snacks of the online workshop

During the Zoom orientation for the online Summer Research Workshop we hinted that there would be information coming to them soon about snacks, but that is all we told them. Then, later, when they received an email alerting them that they had been gifted a build-your-own box of snacks at SnackMagic, they were delighted. The gift had a set amount budgeted for each person, and they could choose whatever snacks they wanted within that amount. Then a box would appear on their doorstep ahead of the Workshop, so they were stocked with snacks for the whole ten days.

The first year we sent these boxes, the Scholars told us to tell future cohorts to mail themselves their snacks at their work addresses rather than at home, so their kids wouldn’t get to the boxes ahead of them!

We chose SnackMagic after a comparison of a few companies like this because the price point was within our expectation, the snack options were diverse (they even have vegan snacks), and the offer drinks, too. One of the Scholars one year chose nothing but drinks, to try new things.

Every day of the Workshop, during a break, we’d compare notes about which snacks we’d selected, which ones we preferred, and which one we were snacking on at that moment.

We ordered a box ahead of time for quality control. I was excited to receive my box.

The cost of snacks for 25 participants is $1,000 ($40 per box, for 25 people). No grant funds were used to pay for these snacks; the library Dean paid for them.

[Edited 8/15/2025 to include the costs of the awards and prizes]

Part of a series of reflections on some of the affective components of the Institute for Research Design in Librarianship (IRDL) program, things I put in place to make the Scholars feel welcomed and delighted in the learning environment.

Awards and prizes in the in-person workshop

To encourage participation during the lecture parts of the summer research workshop, the instructors embedded questions, quizzes, and group activities into their presentations. While the workshop was in person, the instructors would award small prizes for the participants brave enough to answer a question, jump in to share an opinion, or offer a reflection. I bought a bunch of prizes at the Dollar Store ahead of the workshop and put them in a basket so the instructors could easily see all of them. The instructors could choose to award a big sticker, a sheet of stickers, a puzzle, a frisbee, a travel game, etc., depending on how “big” the question was that the participant answered. Once the participants saw what the prizes were in the basket, they’d answer a question in order to make a request for a specific prize.

The prizes met the program design goal we were after, to make participating fun and silly, with a low threat. The instructors even awarded prizes to wrong answers if they were especially good or funny.

Here are a few of the prizes:

A screenshot from an old tweet from a participant that won a puzzle.

Crossword puzzle book prize

A tattoo sleeve prize

In addition, there were group prizes, of which the favorite was for the sampling visualization challenge, where a group of participants needed to work together to use available items in the room to demonstrate an assigned sampling technique. Here’s a photo of one of the group awards:

A photograph of the group that won a group prize of stick-on mustaches, for their efforts in visualizing the simple random sampling technique.

The awards and prizes for the in-person workshop were purchased from the Dollar Store and Amazon, for a cost of around $100.

Awards and prizes in the online workshop

A consideration when we moved the 10-day workshop to an online environment was the amount of time spent synchronously on Zoom. Even with plenty of scheduled and spontaneous screen breaks, asking the Scholars to maintain focus on lectures and exercises seemed like a big request.

A small reward for the Scholars to keep their eyes on the PowerPoint slides was a challenge I designed, named Hidden Iggys. This was the premise:

LA is very close to Disneyland and here at LMU we are familiar with all things Disney. One thing you can do at Disneyland is look for Mickey-Mouse shaped objects throughout the theme park, called “Hidden Mickeys” (https://en.wikipedia.org/wiki/Hidden_Mickey). They’re just for fun, to see how close you’re paying attention as you go through the park.

IRDL has taken that idea and has hidden some images of the LMU mascot lion, named Iggy, in the slide decks. If you spot an Iggy in the slide deck during the lecture, then fill out the Google form telling us where you saw it and get entered into a raffle to win an LMU t-shirt. The Iggys can be large, they can be small, they can be distinct, they can be blurred, and they may be different colors.

There are 2 Iggys each day with lectures, for a total of 18 Iggys. Your name will be entered into the raffle for each Iggy you find. Get them all!

FILL OUT THE FORM FOR EACH IGGY YOU SPOT

All the Iggys will have this same basic shape:

The Google form requested the title of the slide deck where they spotted the Iggy, the slide number, and the Scholar’s name. The Scholars had access to the slides, so they could look for the Iggys there if they didn’t notice them during the lecture session.

After the last lecture section, I used a random number generator to choose one of the entries from the form. The winner got to select from LMU’s online store a branded item to be sent to them in the mail.

Here’s a screenshot of one of the slides with an Iggy “hidden” on it:

A hidden Iggy on a lecture slide

The Hidden Iggy prize cost around $25. The group prize for collaborative demonstration of a sampling technique cost $25 ($5 gift card each, for 5 Scholars).